Build: Improve E2E tests with ESLint rules #29041

JReinhold · 2024-09-03T19:50:24Z

What I did

Initially I fixed some flaky tests by converting to auto-retrying assertions.

Then I realised this this the third time I do this, so I installed and configured eslint-plugin-playwright to hopefully ensure it won't happen again. This required a lot of fixing of all the E2E tests, so I did that too.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

stories
unit tests
integration tests
end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

Add or update documentation reflecting your changes
If you are deprecating/removing a feature, make sure to update
MIGRATION.MD

Checklist for Maintainers

When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts
Make sure this PR contains one of the labels below:
Available labels
- bug: Internal changes that fixes incorrect behavior.
- maintenance: User-facing maintenance tasks.
- dependencies: Upgrading (sometimes downgrading) dependencies.
- build: Internal-facing build tooling & test updates. Will not show up in release changelog.
- cleanup: Minor cleanup style change. Will not show up in release changelog.
- documentation: Documentation only changes. Will not show up in release changelog.
- feature request: Introducing a new feature.
- BREAKING CHANGE: Changes that break compatibility in some way with current major version.
- other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	76.5 MB	76.5 MB	0 B	2.43	0%
initSize	161 MB	161 MB	8.44 kB	-0.26	0%
diffSize	84.7 MB	84.7 MB	8.44 kB	-0.67	0%
buildSize	7.48 MB	7.48 MB	0 B	-1.39	0%
buildSbAddonsSize	1.62 MB	1.62 MB	0 B	2.64	0%
buildSbCommonSize	195 kB	195 kB	0 B	-	0%
buildSbManagerSize	2.31 MB	2.31 MB	0 B	-3	0%
buildSbPreviewSize	352 kB	352 kB	0 B	-	0%
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	4.47 MB	4.47 MB	0 B	2.44	0%
buildPreviewSize	3 MB	3 MB	0 B	-1.46	0%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	6.3s	18.3s	12s	0.58	65.7%
generateTime	18.3s	20.3s	1.9s	-0.49	9.8%
initTime	14.6s	17s	2.4s	-0.5	14.1%
buildTime	11.9s	11.2s	-690ms	-0.8	-6.1%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	9.4s	6.8s	-2s -623ms	-0.21	-38.2%
devManagerResponsive	6.3s	4.4s	-1s -848ms	-0.07	-41.3%
devManagerHeaderVisible	1.2s	689ms	-525ms	-0.85	-76.2%
devManagerIndexVisible	1.2s	722ms	-535ms	-0.9	-74.1%
devStoryVisibleUncached	1.7s	896ms	-885ms	-1.9	🔰-98.8%
devStoryVisible	1.2s	734ms	-520ms	-0.78	-70.8%
devAutodocsVisible	1.1s	667ms	-506ms	-0.44	-75.9%
devMDXVisible	900ms	694ms	-206ms	-0.09	-29.7%
buildManagerHeaderVisible	779ms	748ms	-31ms	0.1	-4.1%
buildManagerIndexVisible	815ms	753ms	-62ms	-0.06	-8.2%
buildStoryVisible	816ms	1s	222ms	2.23	🔺21.4%
buildAutodocsVisible	788ms	702ms	-86ms	0.04	-12.3%
buildMDXVisible	709ms	623ms	-86ms	-1.1	-13.8%

Greptile Summary

This PR improves the E2E tests for Storybook by implementing ESLint rules for Playwright tests and updating test files to use more robust practices.

Added eslint-plugin-playwright and configured rules in code/.eslintrc.js
Updated E2E test files to use auto-retrying assertions, reducing flaky tests
Removed unnecessary 'await' keywords and improved selector specificity
Refactored test structure in framework-specific tests (Next.js, Svelte) for better organization
Updated JSON file tests to use stricter assertions with toStrictEqual()

nx-cloud · 2024-09-03T19:54:31Z

☁️ Nx Cloud Report

CI is running/has finished running commands for commit bb3301c. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

✅ Successfully ran 2 targets

Sent with 💌 from NxCloud.

yannbf · 2024-09-04T06:17:13Z

code/addons/test/package.json

@@ -8,14 +8,14 @@
    "vitest",
    "testing"
  ],
-  "homepage": "https://github.com/storybookjs/storybook/tree/next/code/addons/vitest",


greptile-apps

_{19 file(s) reviewed, 3 comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps · 2024-09-04T09:28:28Z

code/e2e-tests/addon-docs.spec.ts

+/* eslint-disable playwright/no-conditional-expect */
+
+/* eslint-disable playwright/no-conditional-in-test */


style: These ESLint disables might be masking potential issues in the test code. Consider addressing the underlying problems instead of disabling the rules.

greptile-apps · 2024-09-04T09:37:51Z

code/e2e-tests/composition.spec.ts

    ).toBeVisible();

    await page.locator('[id="storybook\\@7\\.6\\.18_components-badge"]').click();
    await expect(
-      await page.locator('[id="storybook\\@7\\.6\\.18_components-badge--default"]')
+      page.locator('[id="storybook\\@7\\.6\\.18_components-badge--default"]')
    ).toBeVisible();

    // Expect composed stories `to be available in the search


syntax: There's a backtick in the comment that should be removed

greptile-apps · 2024-09-04T10:11:48Z

code/e2e-tests/tags.spec.ts

-
-    const noAutodocsAnchor = await root.locator('#anchor--core-tags-remove--no-autodocs').all();
-    expect(noAutodocsAnchor.length).toBe(0);
+    // Autodocs should include autodocs and exclude dev, test


logic: This comment is incorrect. It should be 'Autodocs should include no-dev, no-test and exclude no-autodocs'

JReinhold added 4 commits September 3, 2024 20:23

use auto-retrying assertions in e2e tests

8228120

fix urls in addon-test package.json

01216e1

fix tags e2e test

53ad1a1

add playwright eslint plugin, fix tests by rules

bb3301c

JReinhold self-assigned this Sep 3, 2024

JReinhold added build Internal-facing build tooling & test updates ci:daily Run the CI jobs that normally run in the daily job. labels Sep 3, 2024

yannbf reviewed Sep 4, 2024

View reviewed changes

yannbf approved these changes Sep 4, 2024

View reviewed changes

JReinhold added ci:daily Run the CI jobs that normally run in the daily job. and removed ci:daily Run the CI jobs that normally run in the daily job. labels Sep 4, 2024

JReinhold marked this pull request as ready for review September 4, 2024 09:13

JReinhold merged commit ab66906 into next Sep 4, 2024
115 of 117 checks passed

JReinhold deleted the jeppe/fix-e2e-flake branch September 4, 2024 09:15

github-actions bot mentioned this pull request Sep 4, 2024

Release: Prerelease 8.3.0-beta.3 #29039

Merged

9 tasks

greptile-apps bot reviewed Sep 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build: Improve E2E tests with ESLint rules #29041

Build: Improve E2E tests with ESLint rules #29041

JReinhold commented Sep 3, 2024 •

edited by greptile-apps bot

Loading

nx-cloud bot commented Sep 3, 2024 •

edited

Loading

yannbf Sep 4, 2024

greptile-apps bot left a comment

greptile-apps bot Sep 4, 2024

greptile-apps bot Sep 4, 2024

greptile-apps bot Sep 4, 2024

		/* eslint-disable playwright/no-conditional-expect */

		/* eslint-disable playwright/no-conditional-in-test */

Build: Improve E2E tests with ESLint rules #29041

Build: Improve E2E tests with ESLint rules #29041

Conversation

JReinhold commented Sep 3, 2024 • edited by greptile-apps bot Loading

What I did

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

Manual testing

Documentation

Checklist for Maintainers

🦋 Canary release

Greptile Summary

nx-cloud bot commented Sep 3, 2024 • edited Loading

☁️ Nx Cloud Report

yannbf Sep 4, 2024

Choose a reason for hiding this comment

greptile-apps bot left a comment

Choose a reason for hiding this comment

greptile-apps bot Sep 4, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 4, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 4, 2024

Choose a reason for hiding this comment

JReinhold commented Sep 3, 2024 •

edited by greptile-apps bot

Loading

nx-cloud bot commented Sep 3, 2024 •

edited

Loading